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After  the  initial  tabulation,  which  includes  various  frequency  distributions 
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obviously  skipped  excessive  numbers  of  items,  is  decided.  An  important 
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support.  Index  k*,  which  was  introduced  on  an  earlier  stage  of  this 
research,  was  also  used  for  this  purpose. 
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I  Introduction 


There  have  been  many  research  projects  in  the  history  of 
psychology,  in  which  researchers  attempt  to  measure  mental 
abilities  by  collecting  data  through  the  administration  of  a 
battery  of  tests.  Even  before  the  American  Psychological 
Association  adopted  the  concepts  of  reliability  and  validity 
coefficients  in  1954,  classical  test  theory,  with  all  its 
variations,  has  practically  the  sole  theoretical  foundation  for 
those  research  projects,  and  for  a  long  time  it  has  been  the 
theory  of  mental  measurement. 

There  are  many  deficiencies  and  limitations  in  classical 
mental  test  theory,  however.  The  weakest  points  in  classical  mental 
test  theory  may  be:  1)  it  heavily  depends  upon  a  specific 
population,  or  group,  of  examinees,  and  defines  the  characteristics 
of  a  test,  or  a  test  item,  in  relation  with  a  specific  group  of 
examinees  by  naming  them  as  if  they  were  sole  properties  of  a  test, 
or  a  test  item,  while  in  reality  they  are  the  products  of  the 
interaction  between  the  test,  or  the  test  item,  and  the  group  of 
examinees;  and  2)  it  uses  a  single  measure,  the  correlation 
coefficient,  for  its  important  concepts  such  as  the  reliabilty  and 
validity  of  a  test  and  treats  it  as  if  it  were  a  magic  number, 
while  the  fact  is  that  many  complexities  are  ignored  and  distorted 
because  of  its  use.  Thus  many  researchers  have  misled  and  lost 
themselves  in  the  cobweb  of  classical  mental  test  theory  without 
being  aware  of  the  fact  that  they  have  produced  nothing  but 
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artifacts  (cf.  Samejima,  1977  a). 

Modem  mental  test  theory,  or  latent  trait  theory,  on  the 
other  hand,  is  based  upon  sounder  axioms  and  rationale,  and  has 
gradually  been  invading  the  realm  of  classical  mental  test  theory 
in  the  past  decade,  in  the  area  of  applied  mental  measurement. 

A  relatively  few,  more  sophisticated  researchers  have  started 
conducting  their  research  projects  on  mental  measurement  which  is 
based  upon  latent  trait  theory.  Among  them,  Rasch  model  (Rasch,  1960) 
may  be  the  most  popular  model,  because  of  its  mathematical  simplicity 
and  easiness.  When  they  deal  with  data  in  which  multiple-choice 
test  items  are  used,  however,  most  researchers  have  turned  to  the 
three-parameter  normal  ogive,  or  logistic,  model  (Bimbaum,  1968), 
which  is  based  upon  the  knowledge  or  random  guessing  principle 
in  the  context  of  latent  trait  theory.  It  is  assumed  that  the 
examinee  either  knows  the  answer  to  a  given  multiple-choice  test 
item,  or  guesses  randomly,  in  selecting  one  of  the  given  alternative 
answers.  Thus  the  conditional  probability,  given  ability,  with  which 
the  examinee  answers  the  item  correctly  is  greater  than  the  one  with 
which  the  examinee  knows  the  answer.  When  we  consider  this  conditional 
probabilty  as  a  function  of  ability,  it  is  called  the  item  characteristic 
function  (Lord  and  Novick,  1968).  In  the  three-parameter  normal 
ogive,  or  logistic,  model,  this  item  characteristic  function  for  a 
multiple-choice  item  is  strictly  increasing  in  ability,  but  its 
slope  is  less  than  the  one  for  the  item  characteristic  function 
in  the  normal  ogive,  or  logistic,  model  for  the  free-response 
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test  item.  A  bibliography  of  applied  research  based  upon  the 
three-parameter  models  is  presented  in  Appendix  1  ,  together  with 
one  for  theoretical  works  concerning  the  three-parameter  models, 
which  consists  of  papers  selected,  mainly,  from  recent  issues  of 
leading  journals. 

In  spite  of  the  popularity  of  the  three-parameter  logistic 
model,  except  for  Lord  (Lord,  1970),  the  researchers  rarely  have 
tried  to  validate  the  model  in  relation  to  their  own  data;  they 
simply  adopted  the  model,  and  assumed  its  validity.  Such  attitudes 
of  researchers  cannot  be  considered  as  being  scientific,  and  it  is 
imperative  that  more  people  should  turn  their  attention  to  model 
validation,  or  to  the  search  for  a  suitable  model,  or  models,  before 
they  adopt  one  for  their  research.  It  was  one  of  her  considerations 
when  the  principal  investigator  started  the  series  of  research  on 
theory  and  method  of  estimating  the  operating  characteristics 
without  assuming  any  mathematical  form  several  years  ago  (Samejima, 
1977a,  1977b,  1978a,  1978b,  1978c,  1978d,  1978e,  1978f).  She  has 
also  proposed  a  new  family  of  models  for  the  multiple-choice  test 
items  (Samejima,  1979,  1980),  in  which  she  treats  the  multiple- 
choice  item  as  something  more  than  a  blurred  image  of  the  free-response 
test  item,  which  the  three-parameter  normal  ogive,  or  logistic, 
model  presumes.  Thus  in  these  new  models  each  alternative  wrong 
answer  is  considered  to  be  a  valuable  information  source,  in 
addition  to  the  correct  answer. 


In  the  present  study  of  mental  measurement  based  upon  the 
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analysis  of  the  Iowa  Test  data,  unlike  most  research  projects,  we 
shall  avoid  setting  any  prior  mathematical  models,  and  start  with 
the  careful  examination  of  our  data,  in  order  to  determine  which 
direction  of  research  we  should  take.  In  other  words,  we  are 
intentionally  taking  a  slow  process,  and  by  no  means  are  we 
anxious  to  extract  hasty  conclusions.  This  may  seem  to  be  tedious, 
but  for  truly  scientific  purposes  of  research  this  is  the  fastest 
way,  if  we  wish  to  conduct  research  without  sacrificing  our 
conscience  and  with  the  promise  of  fruitfulness. 

We  hypothesize  two  main  directions  as  our  choices.  One  is 
the  direction  which  leads  to  Informative  Distractor  Model,  and  the 
other  is  the  direction  which  leads  to  Equivalent  Distractor  Model. 
Note,  however,  that  they  are  not  specific  mathematical  models,  but 
a  very  general  categorization  of  ideas  concerning  the  behavior  of 
the  multiple-choice  test  item.  If,  for  instance,  evidence  clearly 
indicates  the  former  direction,  then  we  shall  follow  that  direction 
to  investigate  the  behavior  of  each  test  item  more  specifically. 

If  the  latter  direction  proved  to  be  true,  then  we  shall  head  for 
that  direction  and  investigate  the  specifics  about  each  test  item. 

If  it  turns  out  that  no  single,  general  direction  is  indicated,  then 
we  shall  depend  upon  the  strategy  of  adopting  theory  and  method 
which  enable  us  to  deal  with  test  items  following  either  general 
direction. 

The  present  research  is  only  the  beginning  of  the  analysis 
of  the  Iowa  Test  data,  and  more  research  is  needed  to  add  to  it 


in  the  near  future. 
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II  Tests 

The  battery  of  tests  used  here  is  the  Iowa  Tests  of  Basic 
Skills ,  Form  Levels  9-14 .  These  tests  have  been  designed, 
constructed,  and  revised  at  the  College  of  Education  of  the 
University  of  Iowa  since  1935,  with  the  general  school  population 
in  mind,  and  for  students  of  ages  nine  through  fourteen,  or  grades 
three  through  nine.  All  technical  information  in  this  paper  has 
been  taken  from  either  Form  6  itself  (Hieronymous  and  Lindquist,  1971), 
or  its  Teacher ' s  Manual  (Iowa  Basic  Skills  Testing  Program,  1971) . 

There  are  eleven  tests  in  the  battery,  each  of  which 
focuses  on  a  different  basic  skill.  For  convenience,  hereafter, 
we  shall  call  these  separate  tests  subtests,  in  order  to  avoid 
the  confusion  which  might  occur  when  we  refer  to  both  the  total 
test  battery  and  each  test  in  the  battery.  Following  the  Teacher's 
Manual,  the  descriptions  and  abbreviations  of  these  eleven  subtests, 
together  with  their  administration  schedule  and  working  times,  are 
tabulated  and  presented  in  Table  2-1.  All  the  test  items  are 
power  test  items  with  multiple-choice  format,  with  five  alternative 
answers  for  the  items  in  Subtest  LI,  and  with  four  alternatives 
for  those  in  the  other  ten  subtests.  Within  each  subtest,  test 
items  are  arranged  in  the  ascending  order  of  difficulty.  These 
eleven  subtests  are  designed  to  cover  all  major  areas  of  academic 
interest  for  the  grades  three  through  nine.  The  separate  directions 
for  these  eleven  subtests  are  presented  in  Appendix  II. 


Fourth  Session  30  M_1:  Mathematics  Concepts 

65  Minutes  3®  M-2 :  Mathematics  Problem 

Solving 
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The  numbers  of  test  items  contained  by  the  eleven  separate 
subtests  are  114,  178,  114,  102,  102,  86,  89,  74,  141,  136  and 
96,  respectively,  following  the  order  of  subtests  given  in  Table 
2-1.  They  are  all  presented  in  a  single  test  booklet  of  ninety- 
six  pages.  For  each  of  the  five  levels,  9  through  14,  only  a 
subset  of  each  subtest  is  administered.  For  example,  for  Subtest 
V,  items  1  through  31  are  given  as  Level  9,  items  11  through  48 
are  administered  as  Level  10,  items  24  through  66  are  given  as 
Level  11,  and  so  forth.  Different  answer  forms  for  the 
computerized  scoring  are  provided  for  the  six  different  levels. 

The  standardized  administration  schedule  and  the  working  time 
for  each  subtest  are  presented  in  Table  2-1.  For  the  entire 
test  battery,  the  time  required  for  the  administration  of  each 
level  of  test  is  four  hours  and  thirty-nine  minutes.  It  is 
recommended  that  the  test  be  administered  on  four  consecutive 
days.  It  may  occasionally  have  been  presented  on  four  consecutive 
half-days,  but  never  within  one  day. 

There  are  three  options  available  to  the  classroom  teachers 
in  the  administration  of  the  test.  Among  them,  graded  testing 
is  the  most  common  one,  in  which  a  single  level  of  test  is 
administered  to  all  the  students  in  a  given  classroom.  In  this 
situation,  Level  9  is  given  to  the  third  graders,  Level  10  to 
the  fourth  graders,  Level  11  to  the  fifth  graders.  Level  12  to 
the  sixth  graders,  Level  13  to  the  seventh  graders,  and  Level  14 
to  both  the  eighth  and  ninth  graders.  The  second  option  is 
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out-of-level  testing.  In  this  situation,  the  teacher  selects  an 
optimal  level  for  his  or  her  class,  and  administers  that  level  of 
test,  regardless  of  the  examinees'  nominal  grade.  The  third 
option  is  ind iv idua 1 i z ed  testing,  in  which  each  student  in  the 
class  is  given  the  level  of  test  which  is  most  suitable  for  his 
or  her  level  of  development.  We  notice  that,  even  in  this  third 
situation,  it  is  possible  to  administer  the  test  as  a  group  test 
with  a  single  supervisor,  since  the  administration  schedule  is 
standardized  for  all  levels,  as  we  can  see  in  Table  2-1. 

In  our  data,  only  the  tests  of  Levels  11,  12  and  13  were 
used.  In  most  cases,  the  first  form  of  administration,  i.e., 
graded  testing,  was  adopted.  There  are  a  few  exceptions,  however, 
in  which  the  third  form  was  used.  The  numbers  of  test  items 
contained  in  these  three  levels  of  test  are  461,  487  and  500, 
respectively.  Table  2-2  presents  the  number  of  test  items  in 
each  of  the  eleven  subtests,  for  each  of  the  three  levels.  A 
graphical  representation  is  made  in  Figure  2-1,  to  show  how  these 
three  subsets  of  test  items  in  each  subtest  overlap  among  the 
three  levels.  The  numbers  of  test  items  in  each  subtest,  which 
are  included  in  all  the  three  levels,  in  two  adjacent  levels,  and 
in  single  levels,  are  shown  in  Table  2-3. 


Lach  of  the  r.leven  Suhtests  Administered  to  fach  of 
and  13,  Which  Are  Represented  by  Shaded,  Hollow,  and 
Solid  Bars,  Respectively. 
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III  Data  and  Their  Basic  Tabulations 

Data  were  collected  in  three  different  school  systems  in 
the  State  of  Iowa,  in  the  years  1971  through  1977.  In  their 
original  form,  the  total  number  of  examinees,  including  both  boys 
and  girls,  is  7,581.  Out  of  these  people,  28  students  took  Level  9 
Test  and  114  took  Level  10  Test.  Since  these  are  relatively  small 
numbers,  we  decided  to  exclude  them  from  our  original  group  of 
examinees.  The  other  7,439  examinees  are  classified  into  three 
subgroups,  i.e.,  2,460  students  who  took  Level  11  Test,  2,452  who 
took  Level  12  Test,  and  2,527  who  took  Level  13  Test.  Hereafter, 
we  shall  call  observations  concerning  these  7,439  examinees  the 
original  data. 

Table  3-1  presents  the  frequency  distribution  of  the  test 
items  for  each  of  the  eleven  subtests  with  respect  to  the  percentage 
of  examinees  who  answered  correctly,  for  each  of  Levels  11,  12  and 
13.  We  can  see  that  the  configurations  of  these  eleven  frequency 
distributions  are  alike  across  the  three  levels,  and,  except  for 
Subtest  LI  for  Level  13  and  Subtest  M2  for  Levels  12  and  13,  the 
medians  of  these  frequency  distributions  are  somewhat  higher  than 
50  percent. 

In  these  frequency  distributions ,  all  the  no  responses  were 
treated  as  incorrect  answers.  Table  3-2  presents  the  frequency 
distributions  of  test  items  for  each  of  the  eleven  subtests  with 
respect  to  the  percentage  of  examinees  who  answered  in  one  way 
or  another,  for  each  of  the  three  levels.  We  notice  that,  while 
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Che  majority  of  items,  i.e.,  231  out  of  461  for  Level  11,  319  out 
of  487  for  Level  12,  and  286  out  of  500  for  Level  13,  have  attracted 
some  answers  from  99  percent  or  more  of  the  examinees,  there  are 
some  other  items,  i.e.,  34  for  Level  11,  15  for  Level  12,  and  6  for 
Level  13,  to  which  only  less  than  90  percent  of  the  examinees  have 
responded.  The  cumulative  frequency  of  test  items  to  which  one 
of  the  six  given  percentages,  or  more,  of  the  examinees  responded 
is  presented  in  Table  3-3,  for  each  of  the  eleven  subtests,  and 
for  each  of  the  three  levels. 

The  breakdown  of  the  last  column  of  Table  3-2  into  two 
subcategories,  i.e.,  99.0  to  99.5  percent  and  99.5  to  100.00 
percent,  which  are  added  to  its  right  as  two  additional  columns 
of  Table  3-2,  has  revealed  that,  for  Level  13,  there  are  no  items 
to  which  99.5  percent,  or  more,  of  the  examinees  responded. 

This  result  looks  rather  strange,  and  has  urged  us  to  shift  our 
attention  back  to  our  raw  data.  As  the  result,  it  was  found  out 
that  there  are  a  small  number  of  examinees  who  did  not  respond 
to  a  substantially  large  number  of  test  items.  Table  3-4  presents 
the  frequency  distributions  of  examinees  with  respect  to  the 
number  of  unanswered  test  items  for  Levels  11,  12  and  13.  We 
can  see  in  this  table  that,  while  as  many  as  7,010  examinees  out 
of  7,439  left  only  49  or  less  test  items  unanswered,  there  also 
are  162  examinees  who  did  not  respond  to  as  many  as  100,  or  more, 
test  items.  Our  raw  data  show  there  are  some  examinees  included 


who  skipped  an  entire  subtest,  or  more  than  one  entire  subtest. 
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table  3-3 

Frequency  Distribution  of  Items  Which  Were  Answered, 
Though  Not  Necessarily  Correctly,  by  Specified 
Percentages  of  Examinees  or  More.  (Original  Data) 

Level  11 


Subtest 

Number 
of  Items 

Minimum  1 
902:  92% 

’ercentage  Response 
94%  95%  96%  97% 

V 

- - 

42 

38 

36 

34 

32 

30 

R 

74 

70 

66 

61 

58 

55 

54 

U 

43 

38 

36 

34 

32 

31 

29 

L2 

40 

40 

39 

36 

35 

32 

30 

L3 

40 

40 

40 

40 

40 

40 

36 

W 

32 

32 

32 

32 

32 

32 

32 

VI 

36 

29 

28 

27 

26 

25 

24 

W2 

26 

26 

26 

23 

22 

21 

20 

W3 

56 

42 

40 

37 

35 

34 

31 

MI 

42 

42 

39 

37 

36 

34 

29 

M2 

29 

26 

25 

23 

22 

20 

18 

Total 

461 

427 

409 

386 

372 

356 

333 
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table  3-3  (Continued):  Level  12. 


Subtest 

Number 
of  Items 

Minimum  Percentage 
902  92%:  94%  95% 

Response 
96%  97% 

V 

46 

46 

46 

45 

42 

40 

38 

R 

76 

76 

70 

63 

61 

60 

55 

Ll 

46 

40 

39 

37 

36 

34 

32 

* 

42 

42 

42 

42 

40 

37 

34 

L3 

42 

42 

42 

42 

42 

42 

42 

L4 

32 

32 

32 

32 

32 

32 

32 

W1 

40 

32 

31 

30 

29 

27 

24 

W2 

28 

28 

26 

25 

24 

24 

22 

W3 

59 

59 

57 

53 

50 

49 

47 

Ml 

45 

45 

45 

43 

41 

40 

37 

M2 

31 

30 

29 

27 

26 

25 

24 

Total 

487 

472 

459 

4  39 

423 

410 

387 

TABLE  3-3  (Continued):  Level  13. 


Sub test 

Number 
of  Items 

- — — -  ■  — 

Minimum  Percentage  Response 
90%  92%;  94%  95%  96%  97% 

V 

48 

48 

48 

48 

48 

48 

44 

R 

78 

78 

78 

78 

71 

67 

61 

Ll 

48 

47 

44 

40 

38 

37 

34 

L2 

43 

43 

43 

43 

43 

43 

38 

L3 

43 

43 

43 

43 

43 

43 

43 

L4 

32 

32 

32 

32 

32 

32 

32 

W1 

41 

36 

33 

32 

30 

30 

28 

W2 

28 

28 

28 

28 

26 

25 

23 

W3 

59 

59 

59 

55 

52 

49 

46 

Ml 

48 

48 

48 

48 

45 

43 

40 

M2 

32 

32 

31 

28 

27 

27 

26 

Total 

500 

494 

487 

- - 

475 

455 

444 

415 

|4M*<  AV44I f  » 


TABLE  3-4 


Frequency  Distribution  of  Examinees  with  Respect  to  the 
Number  of  Items  Which  Were  Left  Unanswered  for  Each  of 
the  Three  Levels,  11,  12,  and  13.  (Original  Data) 


Number 

Unanswered 

11 

Level 

12 

13 

Total 

0-49 

2271 

2326 

2413 

7010 

50-99 

120 

85 

62 

267 

100-199 

55 

34 

29 

118 

200-299 

13 

7 

18 

38 

300-500 

1 

_ 

0 

5 

— 

6 

Total 


2460  2452  2527 


7439 
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Table  3-5  presents  the  frequency  distribution  of  such  examinees 
with  respect  to  the  number  of  subtests  which  were  omitted  entirely, 
for  each  of  the  three  levels.  We  can  see  in  this  table  that  one 
examinee  who  did  not  participate  in  the  testing  at  all  is  included 
in  our  group  of  examinees.  The  relationship  between  the  omission 
of  one  or  more  entire  subtests  and  the  number  of  unanswered  items 
is  shown  in  Table  3-6  in  the  form  of  the  frequency  distribution 
of  these  67  examinees  with  respect  to  the  level  and  the  number  of 
unanswered  test  items.  Comparison  of  this  table  with  Table  3-4 
indicates  that  the  six  examinees  who  left  300  or  more  test  items 
unanswered  are  among  those  who  omitted,  at  least,  one  entire 
subtest,  and  as  many  as  27  out  of  38  examinees  who  omitted  200  to 
299  test  items  also  belong  to  this  subgroup  of  examinees.  A 
frequency  distribution  similar  to  Table  3-6  was  made  for  the  126 
examinees  who  left,  at  least,  one  half  of  a  subtest,  but  not  the 
entire  subtest,  unanswered,  and  is  presented  as  Table  3-7.  This 
additional  information  indicates  that,  if  we  exclude  all  the 
examinees  who  left,  at  least,  one  half  of  a  subtest  unanswered 
from  our  total  group  of  examinees,  then  the  number  of  examinees 
who  left  200  or  more  test  items  unanswered  will  become  zero,  and 
only  28  examinees,  who  omitted  more  than  100,  but  less  than  200, 
test  items,  will  be  included.  For  this  reason,  we  have  decided 
to  exclude  these  193  examinees  from  our  original  group  of  examinees 
for  further  analysis.  Hereafter,  we  shall  call  observations 
concerning  the  remaining  7,246  examinees  the  revised  data,  to 
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TABLE  3-6 

Frequency  Distribution  of  Examinees  Who  Omitted,  at 
Least,  One  Entire  Subtest  with  Respect  to  the  Total 
Number  of  Items  Omitted,  for  Each  of  the  Three 
Levels,  11,  12,  and  13.  (Original  Data) 


Number 

Unanswered 

1 

i  11 

1 

Level 

12 

13 

Total 

.  - J 

0-49 

i 

i  ° 

0 

0 

0 

5C-99 

1  2 

5 

5 

12 

100-199 

!  6 

2 

14 

22 

200-299 

I  6 

3 

18 

27 

300-500 

i  1 

0 

5 

6 
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TABLE  3-7 

Frequency  Distribution  of  the  Examinees  Having  Left, 
at  Least,  One  Half  of  a  Subtest  Unanswered,  But  Not 
Omitted  Any  Entire  Subtest,  with  Respect  to  the 
Number  of  Unanswered  Items,  for  Each  of  the  Three 
Levels,  11,  12,  and  13.  (Original  Data) 


Number 

Unanswered 

11 

Level 

12 

13 

Total 

0-49 

9 

1 

3 

13 

50-99 

25 

5 

4 

34 

100-199 

40 

19 

9 

68 

200-299 

7 

4 

0 

11 

300-500 

0 

0 

0 

0 

Total 

81 

29 

16 

126 
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distinguish  themselves  f rora  the  original  data. 

Table  3-8  presents  the  item  identifications  of  the 
fifty-five  test  items,  i.e.,  34  for  Level  11,  15  for  Level  12, 
and  6  for  Level  13,  to  which  less  than  90  percent  of  examinees 
responded  in  the  original  data,  the  percentages  of  examinees  who 
answered  in  one  way  or  another  in  the  original  data,  and  those 
in  the  revised  data.  We  can  see  in  this  table  that  for  most  of 
these  fifty-five  test  items  the  two  percentages  show  a  visible 
improvement  caused  by  the  exclusion  of  the  193  examinees.  The 
frequency  distributions  of  test  items  for  the  eleven  subtests 
with  respect  to  the  percentage  of  examinees  who  answered  in  one 
way  or  another  in  the  revised  data  are  presented  in  Table  3-9, 
for  each  of  the  three  levels.  Comparison  of  this  table  with 
Table  3-2  reveals  a  substantial  improvement  in  the  percentage 
provided  by  the  exclusion  of  the  193  examinees  for  all  the  three 

levels.  Among  others,  we  notice  that  the  frequency  of  test  items 

which  were  answered  by  99  percent,  or  more,  of  examinees  increased 
from  231  to  320  for  Level  11,  from  319  to  350  for  Level  12,  and 

from  286  to  377  for  Level  13. 

Table  3-10  presents  the  frequency  distribution  of  test  items 
for  each  of  the  eleven  subtests  with  respect  to  the  percentage  of 
examinees  who  answered  correctly,  for  each  of  Levels  11,  12  and  13, 
after  the  exclusion  of  the  193  examinees.  There  are  a  substantial 
number  of  categories  whose  frequencies  changed  from  those  in 
Table  3-1,  although  the  two  frequency  distributions  for  each 
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Fifty- five  Test  Items  to  Which  Less  Than  Ninety  Percent 
of  Examinees  Responded  in  One  of  the  Three  Levels  in  the 
Original  Data,  the  Percentages  of  Responses  in  the 
Original  Data,  and  Those  in  the  Revised  Data. 
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requency  Distribution  of  Items  for  Each  of  the  Eleven  Subtests  with  Respect  to  the 
ercentage  of  Examinees  Answering  Correctly,  Each  Interval  of  Percentage  Is  Greater 
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subtest  and  each  of  the  three  levels  are  similar  in  configuration. 

It  should  be  noted  that,  even  in  the  revised  data,  these 
percentages  correct  are  not  independent  from  the  positions  of  the 
test  items  in  each  subtest.  The  relative  frequencies  of  examinees 
who  left  the  items  unanswered  are  given  in  Appendix  III,  in  which 
the  test  items  are  arranged  in  the  order  of  presentation  within 
each  subtest,  for  each  of  the  three  levels.  There  is  a  distinct 
tendency  that  larger  numbers  of  examinees  did  not  respond  to  items 
which  were  presented  later  in  each  subtest.  It  is  obvious, 
therefore,  that,  for  these  later  items,  the  percentage  for  the 
correct  answer  is  less  than  it  should  be  in  the  ideally  set 
free-response  situation. 
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IV  Informative  Distractor  Model  or  Equivalent  Distractor  Model? 

By  Informative  Distractor  Model,  we  mean  the  family  of 
models  in  which  we  assume  the  existence  of  specific  information 
obtainable  from  separate  alternative  answers  ,  including  the 
correct  answer,  of  each  multiple-choice  test  item.  The  family 
of  models,  including  Models  A,  B  and  C,  proposed  by  S  ane j ima 
(Samejima,  1979)  belongs  to  this  general  model. 

In  contrast  to  this  ,  by  Equivalent  Distractor  Model  we 
mean  the  family  of  models  in  which  no  specific  information  is 
expected  from  separate  incorrect  answers,  which  are  given  as 
alternatives  in  the  multiple-choice  test  item.  Thus  all  the 
alternatives,  except  for  the  correct  answer,  of  a  given  multiple- 
choice  item  are  equivalent,  since  the  information  given  by  a 
specific  alternative,  or  distractor,  is  not  different  from  the 
one  given  by  any  one  of  the  remaining  wrong  answers.  The  three- 
parameter  normal  ogive,  or  logistic,  model  belongs  to  this  family 
of  models.  In  this  model,  all  the  information  provided  by  a 
given  wrong  answer  is  pure  noise  resulting  from  random  guessing, 
and,  therefore,  the  alternative  is  equivalent  with  any  one  of 
the  remaining  wrong  answers.  Note,  however,  that  this  type  of 
model  is  not  the  only  one  included  by  the  Equivalent  Distrac tor 
Model.  Suppose  that  the  operating  characteristic  of  each 
wrong  answer  of  a  given  multiple-choice  item  includes  some 
information  about  the  examinee's  ability,  but  all  the  operating 
characteristics,  or  plausibility  curves,  of  the  distractors  are 
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identical.  In  such  a  case,  we  can  say  that  the  item  should 
belong  to  the  Informative  Distractor  Model  in  the  sense  that 
these  distractors  provide  us  with  some  information  concerning 
the  examinee  '  s  ability.  On  the  other  hand,  we  can  also  say  that 
the  item  should  belong  to  the  Equivalent  Distractor  Model,  since 
each  distractor  does  not  have  any  specific  information  which 
distinguishes  itself  from  the  other  distractors.  For  convenience, 
in  the  present  paper,  we  shall  take  the  second  standpoint,  defining 
the  Informative  Distractor  Model  in  the  narrower  sense. 

As  an  indicator  which  enables  us  to  determine  the  direction 
each  test  item  should  take,  we  shall  use  Index  k*  ,  which  was 
proposed  by  Samejima  (Samejima,  1980).  This  index  was  originally 
developed  for  the  purpose  of  invalidating  the  knowledge  or  random 
guessing  principle,  upon  which  the  three-parameter  normal  ogive, 
or  logistic,  model  is  based.  It  is  also  useful  as  the  first  step 
of  searching  the  direction  for  each  test  item,  with  the  two 
general  models  in  mind. 

Let  A  be  the  event  that  the  examinee  does  not  know  the 
answer  to  the  multiple-choice  item  g  .  Vie  shall  consider  the 
probability  space  which  consists  of  the  subpopulation  of  such 
examinees.  The  conditional  probability,  p ( i ] A)  ,  with  which 
the  examinee  selects  the  alternative  i(=l, 2, . . . ,mg  ,  or  m)  of 
item  g  in  this  conditional  probability  space  is  given  by 
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(4.1) 


P (i I A)  J 


-1 


=  p  [  I  p  +  p*] 

1  1VR  1  R 

=  p*t  z  Pi  +  p*]'1  , 

R  i^R  1  R 


i*R 


i=  R 


where  p^  is  the  probability  with  which  the  examinee  chooses  the 
alternative  i  ,  and  p*  is  the  probability  with  which  the  examinee 
guesses  correctly  for  item  g  .  Index  k*  is  defined  in  terms 
of  these  conditional  probabilities,  such  that 


(4.2)  k*  =  exp[-  Z  p(i  |A)  *  log  p  ( i  | A)  ]  »  [  J1  p  (i  jA)P  ^  1 1 

i=l  i=l 

It  is  obvious  that  this  conditional  probability,  p ( i *  A )  ,  for 
a  wrong  answer,  or  i^R  ,  is  proportional  to  p.^  ,  since,  according 
to  the  knowledge  or  random  guessing  principle,  every  examinee  in 
the  original  population  who  has  selected  a  wrong  answer  does  not 
know  the  correct  answer  to  item  g  ,  and,  consequently,  belongs  to 
the  subpopulation  A  .  On  the  other  hand,  we  can  write 


(4.3)  PR  4  PR  » 

for  examinees  who  have  selected  the  correct  answer  R  do  not 
necessarily  belong  to  the  subpopulation  A  . 

Let  9  be  ability,  or  latent  trait,  and  P  (9)  be  c^e 
item  characteristic  function  of  the  multiple-choice  item  g  . 

In  the  three-parameter  normal  ogive,  or  logistic,  model,  this  item 
characteristic  function  is  given  by 
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(4.4)  P  (6)  =  ¥  (0)  +  [l-¥  (6)  ]c  =  c  +  [  1-c  ]¥  (9)  , 

g  g  g  g  g  g  g 

where  ¥  (9)  is  the  item  characteristic  function  of  item  g 
8 

when  it  is  given  as  a  f ree-response  test  item,  which  is  specified 
by 

(4.5)  ¥  (0)  =  (27r)  1/2  f* g(0"V  e-u2/2  du 

8  J  _00 

in  the  normal  ogive  model,  and  by 


(4.6) 


¥  (0)  -  [1  +  exp {-Da  (6-b  )}] 
g  g  g 


in  the  logistic  model,  and  c^  is  the  guessing  parameter  which 

equals  1/m  .  Let  f(0)  denote  the  density  function  of  ability 

6  ,  and  p  and  p  be  the  probabilities  with  which  the  examinee 
“  g 

answers  the  item  correctly  in  the  multiple-choice  situation  and 
the  f ree-response  situation,  respectively.  Thus  we  can  write 


(4.7) 


pg "/  ye)  f(e)  de  • 


and 


(4.8) 


Pp  =  /  P  (9)  f(0)  de  *  p  +  c  (1-p  ) 
R  /  -  g  g  g  g 


It  is  noted  that  the  second  term  of  the  Tightest  hand  side  of  (4.8) 
is  the  probability  with  which  the  examinee  guesses  correctly,  i.e., 
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p*  .  We  also  notice  that  this  quantity  equals  the  probability 
K 

with  which  the  examinee  chooses  a  distractor  i  (^R)  .  Thus  we 
can  write 

(4.9)  p  =  c  ( 1— P  )  =  P*  •  (i^R) 

1  g  g  k 

It  is  obvious  from  (4.9),  (4.1)  and  (4.2)  that  Index  k* 
assumes  its  maximal  value,  m  ,  when  the  three-parameter  normal 
ogive,  or  logistic,  model  holds.  Note,  however,  that  it  is  a 
necessary  condition  for  the  validity  of  the  model,  but  not  a 
sufficient  condition.  Index  k*  can  be  used  for  the  invalidation 
of  the  three-parameter  normal  ogive,  or  logistic,  model,  therefore, 
but  not  for  the  validation  of  the  model,  unless  it  is  combined 
with  other  evidence. 

In  practice,  we  obtain  the  estimate  of  Index  k*  ,  by 

replacing  p(i|A)  in  (4.2)  by  its  estimate,  p(i|A)  .  To  obtain 

this  estimate,  we  can  use  the  frequency  ratio,  ,  as  the 

estimate  of  the  probability,  p^  ,  for  each  wrong  answer,  or 

distractor,  i  (^R)  .  We  notice,  however,  that  the  estimate, 

P*  ,  for  the  probability  p*  ,  is  not  directly  observable  from 

our  data,  and,  in  one  way  or  another,  must  be  defined  indirectly. 

In  congruence  with  the  purpose  of  invalidating  the  knowledge  or 

random  guessing  principle,  such  a  strategy  is  taken  that  we  find 

P*  which  makes  the  resulting  estimate  of  Index  k*  maximal.  In 
R 

so  doing,  we  define  entropy  H*  such  that 
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m  __ 

(4.10)  H*  =  log  k*  =  -  E  p(i|A)*log  p ( i | A)  . 

i=l 


Defining  P*  such  that 


(4.11) 


i^R 


i=R  , 


we  can  write  for  the  estimate. 


p  ( i  I  A) 


such  that 


(4.12)  p(i | A)  =  P*[  E  P*] 

1  i=l 


Then  we  can  rewrite  (4.10)  in  such  a  way  that 


m  .  m  m  m 

(4.13)  H*  =  -[  I  P*J  [  E  P*-log  P*  -  (  E  P*) • log  {  E  P*}] 

s=l  S  i=l  1  1  i=l  1  s=l  s 


We  have  for  the  partial  derivative  of  H*  with  respect  to  P* 

K 

such  that 


(4.14) 


m  „  m  m 

Ml  =  [  E  P*]  [  E  P*-log  P*  -  (  E  P*)-log  P*]  , 

3P*  s=l  s  i=l  1  s=l  S  K 


and,  setting  this  derivative  equal  to  zero,  we  obtain 


P  [  E  P  I 

(4.15)  P*  =  n  P  syR 

R  i^R  1 


or 


log  P*  =  [  E  P  r1  E  P.  -  log  P. 

K  y_  S  /  1  1 

s^R  i^R 


-|-  n  i  ■  Ww n t«  ..  .  .  -? 


(4.16) 
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It  is  obvious  that  Index  k*  introduced  in  the  preceding 
paragraphs  can  be  used  for  our  purpose  of  searching  the  direction 
that  each  multiple-choice  item  should  take,  i.e.,  Equivalent 
Distractor  Model  or  Informative  Distractor  Model.  If  Index  k* 
turns  out  to  be  far  less  than  m  ,  then  wo  must  reject  the 
hypothesis  of  Equivalent  Distractor  Model  for  that  test  item.  If 
it  assumes  a  value  close  to  m  ,  then  we  shall  say  that  Equivalent 
Distractor  Model  should  still  be  under  consideration.  In  both 
cases,  Informative  Distractor  Model  stays  among  the  possibilities. 

It  is  noted  that  the  traditional  chi-square  test  with 
(m-2)  degrees  of  freedom  for  the  goodness  of  fit  for  the  frequencies 
of  the  (m-1)  wrong  answers  with  the  uniform  distribution  as  the 
theoretical  distribution  may  serve  our  purpose  just  as  well, 
without  using  Index  k*.  In  our  pilot  study,  we  applied  it  for 
the  original  data  of  7,439  examinees,  and  the  result  is  summarized 
in  Appendix  IV.  Table  A-4-1  presents  the  frequency  distribution 
of  test  items  for  each  of  the  eleven  subtests  with  respect  to 
the  probability  resultant  from  this  chi-square  test,  for  each  of 
Levels  11,  12  and  13.  As  we  can  see  in  these  tables,  only  23, 

22  and  21  test  items  indicate  the  acceptance  of  the  respective 
uniform  distributions,  or  the  acceptance  of  Equivalent  Distractor 
Model,  for  Levels  11,  12  and  13,  respectively,  even  if  we  take 
as  low  a  level  of  significance  as  0.0005  .  Table  A-4-2  presents 
the  item  identification,  the  probability  and  the  percentage  of 
the  correct  answer  of  each  of  these  66  test  items.  We  can  see 
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in  this  table  tliat  ,  il  we  raise  the  level  o:  signiticanee  te 
0.01,  then  all  the  items  whose  probabilities  are  marked  with  ** 
will  be  excluded,  to  make  the  total  number  of  items  as  small 
as  45,  i.e.,  17  for  Level  11,  11  for  Level  12,  and  17  for  Level  13. 
If,  further,  we  raise  the  level  of  significance  up  to  0.05,  then 
the  total  number  of  items  will  be  reduced  to  36,  i.e.,  15  for 
Level  11,  7  for  Level  12,  and  14  for  Level  13,  excluding  all 
the  items  whose  probabilities  are  marked  with.  *  .  This  number  is 
only  2.5  percent  of  the  total  number  of  items,  1,448.  There  are 
only  19  items  whose  probabilities  are  greater  than  0.2,  i.e.,  R-78, 
Wl-21 ,  Wl-25 ,  W3-39 ,  >11-31,  Ml-38,  Ml-45  and  Ml-68  for  Level  11, 
V-79 ,  R-78,  Wl-29,  W2-44,  W3-53  and  Ml-95  for  Level  12,  and 
V-93 ,  R-140 ,  R-155 ,  W2-44,  and  Ml-107  for  Level  13.  Since  it 
is  unlikely  that  this  result  obtained  upon  the  original  data 
is  substantially  different  from  the  one  obtainable  from  our 
revised  data,  we  must  take  it  as  the  suggestion  for  the  rejection 
of  Equivalent  Distractor  Model.  Note,  however,  that  the  chi-square 
test  applied  for  our  data  has  severe  criteria,  since,  in  most 
cases,  our  sample  sizes  are  very  large.  It  is  interesting  to 
note  that,  except  for  Wl-25  (57.9%)  of  Level  11,  V-79  (46.6%) 
and  Ml-95  (29.4%)  of  Level  12,  and  V-93  (49.9%),  R-140  (33.8%), 
R-155  (35.7%)  and  Ml-107  (40.6%)  of  Level  13,  all  the  other  12 
test  items,  whose  probabilites  are  greater  than  0.2,  have  greater 
percentages  of  the  correct  answer  than  the  medians  of  the  separate 
subtests,  as  is  clear  if  we  compare  Table  A-4-2  with  Table  3-1. 
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Since  che  examinees  who  nswer*  d  correct  iv  were  exc  1 ucco  :  rec 
Chose  used  for  die  ehi-scuare  test,  we  can  see  chat  these  u^t 
items  whose  probabilities  are  .  reater  than  0.?  are  based  upon 
relatively  small  sample  sizes.  As  a  whole,  however ,  sample 
sizes  are  large,  and  the  ohi-s<  uare  test  is  very  sensitive  to 
small  deviations  r«r  tin  assi.r.ed  uniform  distributions. 

In  contrast  to  th»  chi-square  test,  we  can.  see  from  t  .  „"i 
that  the  estimated  Index  k*  is  insensitive  to  tile  sample  size, 
because  the  sampling  fluctuati.  n  participates  in  the  resulting 
estimate  only  through  the  computation  of  the  proportions,  ? 

Thus,  whether  it  is  right  or  vr ong ,  if  we  wish  to  ignore  the 
sampling  luctuations  of  the  proportions,  then  the  values  of 
estimated  Index  k*  can  be  comparable  across  different  sample 
sizes . 

Table  9-1  presents  the  : requency  distribution  of  the 
items  of  each  of  the  ten  subtests,  excluding  Subtest  LI,  which 
consists  of  five-alternative  test  items,  with  respect  to  the 
resultant  values  oi  the  estimated  Index  k* ,  for  each  of  Levels 
11,  12  and  13.  The  corresponding  result  for  Subtest  LI  is 
presented,  separately,  as  Table  9-2,  for  all  the  three  levels. 

We  can  see  in  Table  4-1  hat  tie  con f igurat ions  of  these 
frequencies  are  similar  across  the  three  levels,  with  the  range 
of  the  estimated  Index  k*,  2.25  through  9.00,  for  each  level. 

This  is  also  the  case  with  Subtest  LI,  with  the  range  of  estimated 
Index  k*  2.25  through  9 .  ->0  for  most  items,  as  is  shown  ir. 


requeruy  distribution  ot  Fout  -A  iternat  lv«-  Item.-,  with  kespeii  to  Index  k*  l.>x  F.  ' 
t  the  I en  Subtests.  1  be  Kjnet  j[  Index  k*  j n  redter  1  ban  o t  I.judl  to  the  lower 
fcnd  anJ  Less  ri.an  the  pper  tnd  of  f-scr-  Interval. 
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Frequency  Distribution  of  Five-Alternative  Items  of 
Subtest  LI,  with  Respect  to  Index  k*  ,  for  Levels 
11,  12,  and  13,  Respectively. 
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Table  4-2.  We  notice  in  Table  4-1  that,  for  each  of  the  three- 
levels,  the  mode  of  the  total  frequency  distribution  is  the 
highest  category,  3.75  through  4.00.  If  we  examine  the  frequency 
distributions  of  separate  subtests,  however,  we  will  notice  that 
there  are  some  variations  among  their  configurations.  Above  all, 
it  is  noted  that  Subtests  L2 ,  L3  and  L4  have  different  modes 
from  the  highest  category,  j.o.,  mostly  either  the  category,  3.00 
through  3.25,  or  the  category,  3.25  through  3.50.  This  tendency 
is  also  shared  by  Subtest  LI,  which  has  five-alternative  multiple- 
choice  test  items,  as  is  shown  in  Table  4-2. 

Figure  4-1  presents  the  scatter  diagram  of  all  the  four- 
alternative  test  items  which  are  included  in  both  Levels  11  and 
12,  with  respect  to  the  values  of  the  estimated  Index  k* ,  and 
the  corresponding  scatter  diagram  for  Levels  12  and  13.  The 
numbers  of  the  shared  test  items  are  264  for  Levels  11  and  12, 
and  323  for  Levels  12  and  13.  There  is  a  tendency  in  both 
scatter  diagrams  that  more  items  are  located  below  the  line  of 
45  degrees,  which  is  drawn  in  the  graphs,  and  this  tendency  is 
more  conspicuous  in  the  first  one  in  which  Levels  11  and  12 
are  compared.  This  indicates  the  existence  of  items  whose  values 
of  the  estimated  Index  k*  are  higher  for  the  lower  level  and 
lower  for  the  higher  level.  Similar  scatter  diagrams  were 
drawn  for  each  of  the  eleven  subtests,  including  Subtest  LI, 
and  for  each  of  the  two  pairs  of  levels,  and  the  resultant  twenty- 
two  graphs  are  presented  in  Figures  4-2  and  4-t.  V’c  can  see  in 
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FIGURE  4-1 

Comparison  of  the  Values  of  Index  k*  for  the  Four-Alternative 
Items  of  the  Eleven  Subtests  Administered  to  Two  Adjacent 
Levels  of  Students:  Levels  11  and  12. 
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FIGURE  4-1  (Continued):  Levels  12  and  13. 
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FIGURE  4-2 

Comparison  of  the  Values  of  Index  k*  for  the  Items  of  Each  Subtest 
Administered  to  the  Students  of  Both  Levels  11  and  12. 
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FIGURE  4-2  (Continued):  Levels  11  and  12. 


LEVEL  12  LEVEL  12 


LEVEL  13  LEVEL  13 


-53- 


LEVEL  12 

(57  Items) 


(32  Items) 


FIGURE  4-3 

Comparison  of  the  Values  of  Index  k*  for  the  Items  of  Each  Subtest 
Administered  to  the  Students  of  Both  Levels  12  and  13. 
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FIGURE  4-3  (Continued):  Levels  12  and  13. 
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FIGURE  4-3  (Continued):  Levels  12  and  13. 
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chis  figure  that  the  above  tendency  exists  in  all  the  comparisons. 

Table  4-3  presents  the  identification  of  each  four- 
alternative  test  item  whose  estimated  Index  k*  is  3.9  or  greater, 
the  value  of  the  estimated  Index  k*,  and  the  probability  obtained 
as  the  result  of  the  chi-square  test,  which  was  referred  to 
earlier  in  this  chapter.  Mote,  however,  that  this  probability 
was  obtained  upon  the  original  data,  while  the  estimated  Index  k* 
was  obtained  upon  the  revised  data.  The  numbers  of  four- 
alternative  test  items  whose  estimated  Index  k*  are  greater  than, 
or  equal  to,  3.9  are  78,  73  and  76  for  Levels  11,  12  and  13, 
respectively.  We  can  see  there  is  an  approximate  correspondence 
between  the  probabilities  which  are  greater  than,  or  equal  to, 

0.1  and  the  <stimated  values  of  Index  k*  which  are  3.9  or  greater. 

Eighty-two  examples  of  the  frequency  distribution  of  the 
examinees  with  respect  to  their  choices  of  an  answer  out  of  the 
four  alternatives  are  presented  as  Figure  4-4.  These  test  items 
are  the  first  twenty-four  items  in  each  of  the  three  levels  in 
Table  4-3,  added  by  those  listed  in  the  same  table  whose 
corresponding  probabilities  obtained  as  the  result  of  the  chi- 
square  test  are  greater  than,  or  equal  to,  0.0005  .  Thus  we  have 
26,  27  and  29  four-alternative  test  items  for  Levels  11,  12 
and  13,  respectively,  to  make  the  total  number  of  illustrated 
test  items  82.  In  each  histogram,  also  drawn  by  a  dotted  line 
is  the  estimated  proportion,  P*  ,  multiplied  by  the  number  of 
examinees  who  answered  the  item  in  one  way  or  another,  or  the 
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total  number  of  examinees  subtracted  by  the  number  of  those 
who  did  not  answer  the  item  at  all.  We  can  see  in  this  figure 
that  most  of  cnese  hi  t  ••'grams  i  re  lr'St  •  ■  rectangles,  if  we 
replace  the  frequency  for  the  correct  answer  by  the  height 
indicated  by  the  dotted  line  in  each  histogram.  It  is  especially 
true  if  we  focus  our  attention  those  items  whose  estimates  of 
Index  k*  are  greater  t nan  3. ,  i.e.  ,  V-61,  R-78,  Wl-21,  Wl-25, 
Wl-29,  W3-39 ,  W3-40,  W3-5-,  Ml -31  .  "1-37,  >11-38,  Ml-45  and  Ml-68 
of  Level  11,  V-57 ,  V-79,  R-78,  Wl-29,  Wl-53,  W2-44 ,  W3-33,  >’1-74 
and  Ml-95  of  Level  12,  and  V-61,  V-79,  V-93,  R-131,  R-140,  R-142, 
R-155 ,  Wl-78,  W2-44 ,  W2-48,  W3-83,  >11-107  and  >12-75  of  Level  13. 
On  the  other  hand,  if  we  shift  our  attention  to  those  whose 
values  of  the  estimated  Index  k*  are  relatively  low,  like  W2-28 
of  Level  11  (k*=3. 97392) ,  W3-83  of  Level  12  (k*=3. 97637)  and 
M2-78  (k*=3. 97137)  of  Level  13,  we  shall  find  soma  substantial 
deviations  from  rect angularity .  For  the  croup  as  a  whole, 
however,  these  histograms  are  close  enough,  to  rectangles,  as  we 
expect  from  the  high  values  of  estimated  Index  k*. 

It  should  be  noted  that  among  these  eighty-two  four- 
alternative  test  items  we  find  no  items  rrom  Subtest  L2,  L3  or 
L4.  These  subtests  are  all  developed  for  measuring  language 
skills.  In  fact,  in  the  total  group  of  227  test  items,  whose 
values  of  the  estimated  Index  k*  are  greater  than  3.9,  there 
are  only  four  test  items  :  rc~.  ti.i  <.  ; :  in  a, .to  t  • ,  i.e., 

L 2  —  3 8  (k*=3. 95473)  an:  L!-~°  '  vel  II,  and 
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L3-49  (k*=3. 95658)  and  L2-58  (  k*=  3 . 95  3 1 8  ,i  oi  Level  12.  wnicn 
are  actually  two  items  shared  by  both  Levels  11  and  12.  This 
result  is  no  surprise,  considering  the  fact  that  for  these 
subtests  the  modes  of  the  estimated  Index  k*  are  much  lower 
than  those  of  the  other  subtests,  as  we  have  seen  earlier  in 
Table  4-1.  A  close  examination  of  the  contents  of  the  test 
items  of  these  four  subtests,  including  Subtest  Ll ,  and  their 
results  of  analysis  reveals  the  following  facts. 

(1)  All  the  questions  in  these  four  language  skill 
subtests  are  in  the  form  of  having  the  examinee  find 
mistakes  in  spelling,  cap  it tlization,  punctuation 
and  usage,  respectively. 

(2)  Unlike  the  test  items  in  the  other  seven  subtests, 
these  items  have  "No  mistakes"  as  the  last  alternative, 
and  for  most  items  this  alternative  has  a  high  frequency, 
even  when  it  is  a  wrong  answer. 

From  these  facts  and  the  above  results,  it  is  obvious  that 
Equivalent  Distractoi  Model  is  not  suitable  for  the  items  of 
the  four  subtests  of  language  skills,  including  Subtest  1.1, 
which  consists  of  five-alternative  test  items.  For  these  items, 
Informative  Distractor  Model  may  be  more  appropriate. 

It  should  also  be  noted  that,  except  for  a  few  items 
like  V-61  and  '‘1-63  of  Level  11,  Ml-95  and  M2-60  of  Level  12, 
and  R-140,  R-155,  Vl-75,  Wl-78,  W2-67,  M2-60,  M2-75  and  M2-82 
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of  Level  13,  these  items  presented  in  Figure  4-4  are  relatively 
easy  items  for  the  respective  groups  of  examinees.  In  such  a 
case,  we  cannot  expect  this  type  of  frequency  distribution  to 
reveai  the  information  each  alternative  has,  even  if  the  item 
belongs  to  Informative  Distractor  Model.  If  we  exclude  all 
the  four-alternative  test  items  which  belong  to  one  of  the 
three  subtests  of  language  skills,  there  are  76  items  out  of 
306  for  Level  11,  71  items  out  of  325  for  Level  12,  and  76  out 
of  334  for  Level  13,  whose  estimates  of  Index  k*  are  3.9  or 
greater.  These  numbers  are  interpreted  as  24.8,  21.8  and  22.8 
in  percentage  for  Levels  11,  12  and  13,  respectively,  for  the 
remaining  seven  subtests.  They  are  by  no  means  large  numbers, 
however . 

Table  4-4  presents  the  item  identification,  the  value  of 
estimated  Index  k* ,  and  the  probability  obtained  by  the  chi-square 
test  for  each  of  the  four-alternative  test  items  whose  estimates 
of  Index  k*  are  the  lowest,  i.e.,  3.0  or  less.  These  items 
number  134,  of  which  42  are  of  Level  11,  49  are  of  Level  12,  and 
the  remaining  43  arc  of  Level  13.  We  can  sec  in  this  table  that 

all  the  probabilities,  which  were  obtained  upon  the  original 
data,  are  less  than  0.0005  .  It  is  noted  that  there  are  many 
items  from  Subtest  L2 ,  L3.  or  L4,  unlike  those  for  the  highest 
Index  k*  group,  which  are  shown  in  Table  4-3.  In  fact,  there 
are  as  many  as  27  such  items  for  Level  11,  31  for  Level  12  and 
26  tor  Level  13.  Out  of  those  tost  items,  7,  11  and  9  items 
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have  the  values  of  Che  estimated  Index  k*  less  than,  or  equal 
to,  2.6,  respectively,  for  Levels  11,  12  and  13.  Histograms 
similar  in  nature  as  those  in  Figure  4-4  are  drawn  for  the 
frequency  distributions  of  these  27  four-alternative  test  items, 
and  are  presented  as  Figure  4-5.  We  can  see  that  these  histograms, 
whose  frequencies  for  the  correct  answer  are  replaced  by  the 
dotted  lines,  are  by  no  means  close  to  rectangles.  There  is  no 
reason  to  accept  Equivalent  Distractor  Models  for  these  test 
items.  Among  them,  there  are  such  items  as  R-34  and  R-50  for 
Level  11,  L2-79  and  L3-72  for  Level  12,  and  L2-79  and  L3-72 
for  Level  13  whose  values  of  P*  multiplied  by  the  respective 
numbers  of  examinees  who  answered  in  one  way  or  another  exceed 
the  corresponding  frequencies  for  the  correct  answer;  in  these 
cases,  the  use  of  Index  k*  itself  is  meaningless. 

Figure  4-6  presents  the  frequency  distributions  of  six 
four-alternative  test  items  for  each  of  the  three  levels,  which 
were  arbitrarily  selected  from  those  whose  values  of  the 
estimated  Index  k*  are  greater  than,  or  equal  to,  3.0  and  less 
than  3.6.  They  are  samples  from  the  four-alternative  test 
items  with  intermediate  values  of  the  estimated  Index  k*.  We 
can  see  that  these  histograms,  in  which  the  frequencies  for 
the  correct  answer  are  replaced  by  the  dotted  lines,  are  also 
far  from  rectangles.  They  also  include  one  item,  i.e.,  L4-74 
for  Level  13,  for  which  the  dotted  line  exceeds  the  frequency 
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for  the  correct  answer. 

As  an  additional  information.  Figure  '«-/  presents  sir 

‘Histograms  for  W 1  - 1  >  and  K-Jh  o:  -••••■•'>  ;  ■  1  •  e  L*,c  *4 

cbt  lined  by  the  chi-square  test,  w:.:cn  v.is  _o:.<-uetec3 
original  data,  turned  out  t-  he  0.005  am;  0.001,  respe-.*  . 
although  their  values  oi  est  invitee  1  n«.h-x  i-e-  are  less  t.m..  -• 

i.e.  ,  J.87  789  for  W 1  -  1  and  5.8  1U'9  for  K-Jc  . 
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V  Comparisons  of  the  Results  of  Test  Items  Which  Were  Administered 
to  Levels  11,  12,  and  13 

There  are  certain  test  items  which  are  included  in  all  the 
three  levels.  They  are  items  58  through  66  of  Subtest  V,  80 
through  98  of  Subtest  R,  58  through  66  of  Subtest  LI,  49  through 
58  of  Subtest  L2,  49  through  58  of  Subtest  L3,  44  through  54  of 
Subtest  L4,  38  through  47  of  Subtest  Wl,  41  through  46  of  Subtest 
W2,  and  67  through  82  of  Subtest  W3.  Thus  there  are  nine  items  of 
Subtest  V,  which  were  administered  to  all  the  three  groups  of 
students,  nineteen  of  Subtest  R,  nine  of  Subtest  LI,  ten  of  Subtest 
L2,  ten  of  Subtest  L3,  eleven  of  Subtest  L4,  ten  of  Subtest  Wl, 
six  of  Subtest  W2  and  sixteen  of  Subtest  W3,  which  make  the  total 
number  of  test  items  shared  by  all  the  three  levels  of  test  100. 
There  is  no  item  which  is  included  in  all  three  levels  for  Subtests 
Ml  and  M2. 

It  is  evident  that,  for  the  behavior  of  the  test  item  to 
follow  Equivalent  Distractor  Model,  not  only  the  value  of  estimated 
Index  k*  should  be  cl'.3e  to  m  for  one  level  of  examinees  but  also 
for  all  three  levels.  For  this  reason,  it  will  be  worthwhile  to 
compare  the  results  across  the  three  levels  for  these  one  hundred 
test  items  which  are  included  in  all  the  three  levels  of  test. 

Tables  5-1  and  5-2  present  the  three  values  of  estimated 
Index  k*  ,  which  were  rounded  to  the  second  digit  after  the  decimal 
point,  for  each  of  the  ninety-one  four-alternative  test  items,  and 
for  each  of  the  nine  five-alternative  test  items,  respectively. 

We  can  see  that  only  seven  four-alternative  test  items,  i.e.,  V-61, 


TABLE  5-1  (Continued) :  Subtest  R. 
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11X12X13)  I  11 


St  20  1 
57  21  2 
SB  22  3 

SB  23  t 

60  2*  5 

61  23  6 

62  26  7 

63  27  8 

64  28  9 

65  29  10 

66  30  11 

67  31  12 

68  32  13 


73  37  18 
76  38  19 


3.86 

3.83 

3.3S 

3.17 

3.16 

2.87 

3.10 

2.82 

3.61 

3.39 

3.48 

3.46 

3.08 

3.12 

3.30 

2.77 

3.98 

3.96 

3.74 

3.63 

3.90 

3.86 

3.65 

3.53 

3.91 

3.96 

3.74 

3.67 

3.61 

3.33 

3.82 

3.72 

3.77 

3.72 

3.81 

3.74 

3.75 

3.83 

TABLE  5-1  (Continued);  Subcaat  L2, 
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(11X12X13) 


40  2(7  10 


I 

:«a  Nu*b«r* 

Level 

(11X12X13) 

11 

12 

13 

49 

31 

U 

1 

3.95 

3.96 

3.86 

50 

32 

12 

2 

3.75 

3.72 

3.80 

51 

33 

13 

3 

3.80 

3.61 

3.36 

52 

34 

14 

4 

2.94 

2.90 

2.97 

53 

35 

IS 

5 

2.9t 

2.93 

2.72 

54 

3t 

It 

6 

3.10 

3.05 

2.99 

55 

37 

17 

7 

3.49 

3.20 

2.82 

56 

38 

18 

8 

3.62 

3.70 

3.80 

57 

39 

19 

9 

3.58 

3.61 

3.58 

58 

40 

20 

10 

3.28 

3.30 

3.33 

Item  Numbers 


(11X12X13) 


22 

12 

1 

23 

13 

2 

24 

14 

3 

23 

13 

4 

26 

16 

3 

27 

17 

6 

28 

18 

7 

29 

11 

8 

30 

20 

9 

31 

21 

10 

32 

22 

11 

Subtest 

L4. 

Lmvel 

12 

3.00 

■si 

3.33 

3.76 

■89 

3.61 

3.46 

3.18 

3.17 

3.28 

3.34 

3.07 

2.83 

3.42 

3.33 

3.78 

3.75 

3.63 

3.73 

3.41 

3.39 

TABLE  5-1  (Continued):  Subtest  VI. 
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!  Level 

rirn 

12 

13 

38 

27 

11 

1 

3.91 

39 

28 

12 

2 

3.65 

40 

29 

13 

3 

3.92 

41 

30 

14 

4 

3.71 

42 

31 

13 

3 

3.37 

43 

32 

16 

6 

3.96 

44 

33 

17 

7 

3.84 

45 

34 

18 

8 

3.96 

46 

33 

19 

9 

3.93 

47 

36 

20 

10 

3.94 

TABLE  5-1  (Continued):  Subtcst  W2. 


TABLE  5-1  (Continued):  Subtest  U3. 


Item  Numbers 
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41 

21  9  1 

3.97 

42 

22  10  2 

3.81 

43 

23  11  3 

3.94 

44 

24  12  4 

3.91 

45 

23  13  3 

3.93 

46 

26  14  6 

3.63 

(liyi2X13) 

11 

12 

13 

67 

41  23  1 

3.34 

3.07 

2.78 

68 

42  26  2 

3.00 

2.75 

2.68 

69 

43  27  3 

3.92 

3.77 

3.67 

70 

44  28  4 

3.93 

3.92 

3.97 

71 

45  29  3 

3.34 

3.38 

3.33 

72 

46  30  6 

3.25 

2.99 

3.10 

73 

47  31  7 

3.80 

3.56 

3.57 

74 

48  32  8 

3.79 

3.78 

3.75 

75 

49  33  9 

3.37 

3.13 

3.22 

76 

SO  34  10 

3.20 

3.32 

3.24 

77 

51  35  11 

3.63 

3.38 

3.36 

78 

52  36  12 

3.92 

3.76 

3.83 

79 

33  37  13 

3.71 

3.63 

3.57 

80 

34  38  14 

3.49 

3.46 

3.53 

81 

55  39  13 

3.41 

3.06 

3.18 

82 

36  40  16 

3.71 

3.54 

3*44 
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TABLE  5-2 

Three  Different  Values  of  Index  k*  for  Each  of  the  Five 
Alternative  Items  That  Were  Administered  to  All  Three 
Levels,  with  Their  Respective  Item  Numbers. 

Subtest  LI 


1  Item  Numbers 

Level 

t 

11 

12 

13 

58 

35 

19 

1 

2.36 

2.39 

59 

36 

20 

2 

3.03 

2.92 

60 

37 

21 

3 

4.17 

3.99 

3.87 

61 

38 

22 

4 

3.79 

3.62 

3.33 

62 

39 

23 

5 

2.90 

2.83 

2.67 

63 

40 

24 

6 

4.31 

4.14 

4.00 

64 

41 

25 

7 

3.46 

3.28 

3.20 

65 

42 

26 

8 

3.66 

3.36 

3.31 

66 

43 

27 

9 

4.36 

4.32 

3.94 
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R-88,  Wl-45,  Wl-46,  W2-44,  W2-45  and  W3-70,  have  three  estimates 
of  Index  k*  all  of  which  are  greater  than,  or  equal  to,  3.9  . 

If  we  shift  this  critical  value  from  3.9  to  3.8  ,  these  seven 
four-alternative  test  items  are  joined  by  eleven  more  items,  i.e., 
V-63,  V-66,  R-80,  R-90,  R-92,  L2-58,  L3-49,  Wl-40,  Wl-43,  Wl-47 
and  W2-41  .  There  are  no  five-alternative  test  items  of  Subtest  LI 
which  are  comparable  to  these  eighteen  four-alternative  test  items. 

We  have  selected  Subtests  R  and  W3,  which  have  the  two  largest 
numbers  of  shared  test  items,  as  our  examples,  and  drawn  the  set 
of  three  histograms,  which  are  similar  in  nature  to  those  in 
Figures  4-4,  4-5  and  4-6,  for  the  set  of  three  frequency  distributions 
for  each  item.  Figures  5-1  and  5-2  present  the  resultant  nineteen 
sets  of  histograms  for  Subtest  R,  and  the  sixteen  sets  for  Subtest 
W3,  respectively. 

It  is  interesting  to  note  that  some  items  show  evidence  of 
differential  information  provided  by  separate  wrong  answers. 

For  example,  alternative  4  of  R-80  seems  to  attract  students  of 
intermediate  reading  ability,  while  alternative  1  of  the  same  item 
appears  to  attract  students  of  lower  levels  of  ability.  It  is  clear 
that  many  items  have  one  or  more  effective  dis tractors,  and, 
among  others,  alternative  2  of  R-86  proved  to  be  powerful.  Most 
histograms  have  some  regularities  in  the  way  the  frequencies  change 
across  the  three  levels,  which  suggest  that  the  examinees  selected 
their  answers  intentionally  rather  than  by  random  guessing. 
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FIGURE  5-1 

Comparison  of  Che  Three  Frequency  Distributions  of  Examinees 
with  Respect  to  Their  Choices  of  Alternatives  for  Each  of  the 
Twenty  Items  of  Subtest  R,  Which  Were  Administered  to  All 
Three  Levels  of  Students,  with  the  Estimated  Proportion 
of  Examinees  Guessing  Correctly  (Dotted  Line). 
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FIGURE  5-1  (Continued):  Subtest  R 
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Comparison  of  the  Three  Frequency  Distributions  of  Examinees 
with  Respect  to  Their  Choices  of  Alternatives  for  Each  of  the 
Sixteen  Items  of  Subtest  W3,  Which  Were  Administered  to  All 
Three  Levels  of  Students,  with  the  Estimated  Proportion 
of  Examinees  Guessing  Correctly  (Dotted  Line) . 
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FIGURE  5-2  (Continued):  Subtest  W3. 
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VI-1 


VI  Discussion  and  Conclusions 

We  have  seen  in  the  previous  chapters  that,  generally  speaking, 
evidence  indicates  the  direction  of  Informative  Distractor  Model, 
rather  than  that  of  Equivalent  Distractor  Model.  This  is  especially 
true  with  the  test  items  for  measuring  language  skills,  i.e.,  those 
of  Subtests  LI,  L2,  L3  and  L4.  There  are  certain  test  items, 
however,  for  which  the  direction  of  Equivalent  Distractor  Model  may 
be  more  suitable,  as  we  have  discussed  in  the  preceding  two  chapters. 
For  these  reasons,  it  will  be  wise  to  take  the  strategy  of  adopting 
theory  and  method  which  make  it  possible  for  us  to  handle  the  test 
items  following  either  general  direction. 

In  so  doing,  perhaps  the  most  promising  way  is  to  adopt  the 
methods  and  approaches  for  estimating  the  operating  characteristics 
without  assuming  any  mathematical  form.  The  selection  of  items 
which  can  be  used  as  a  substitute  for  the  Old  Test  will  be  the 
most  crucial  point  in  that  process.  If  it  succeeds,  then  we  shall’ 
be  able  to  conduct  the  item  analysis  in  the  true  sense  of  the  word, 
which  leads  to  the  evaluation  and  suggestions  for  improvement 


of  each  test  item. 
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APPENDIX  II 


Directions  of  Iowa  Tests  of  Basic  Skills 


for  Each  of  the  Eleven  Subtests 
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APPENDIX  III 


Relative  Frequencies  of  Examinees  Who  Left  the  Items 
Unanswered  in  Each  Subtest,  Arranged  in  the  Order 
of  Presentation  of  the  Items 
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APPENDIX  IV 


Frequency  Distribution  of  Items  for  Each  of  the  Eleven 
Subtests  with  Respect  to  the  Probability  Resultant  from 
the  Chi-Square  Test  of  the  Goodness  of  Fit,  and  the 
Percentage  Correct  Response  for  Each  of  Those  Items 
Whose  Resultant  Probability  Is  0.001  or  Greater 


I 

I 


! 


TABLE  A-i-i 


Frequency  Distribution  of  Items  tor  Each  of  the  Eleven  Subtests  with  Respect  to  the 
Probability  Resultant  from  the  Chi-Square  Test  of  the  Goodness  of  Fit.  The  Uniform 
\  Distribution  Is  Assumed  for  the  Theoretical  Frequency  Distribution  for  the  Incorrect 

Alternatives.  Number  of  Degrees  of  Freedom  for  Each  Item  Is  2,  Except  for  the  Items 
of  Subtest  LI  for  Which  the  Number  of  Degrees  of  Freedom  Is  3.  (Original  Data) 

Level  11 
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TABLE  A-4-2 

Percentage  Correct  Response  for  Each  of  the  Items  Whose 
Probability  Obtained  by  the  Chi-Square  Test  of  the 
Goodness  of  Fit  Is  0.001  or  Greater.  (Original  Data) 

Level  li 


Probability 

Percentage 

Correct 

0.061 

35.9 

0.001** 

93.1 

0.081 

92.9 

0.002** 

64.8 

0.889 

79.2 

0.004** 

58.2 

0.063 

81.1 

0.005** 

93.7 

0.036* 

90.7 

0.824 

83.9 

0.514 

57.9 

0.147 

52.1 

0.012* 

85.7 

0.001** 

63.7 

0.565 

63.7 

0.111 

76.8 

0.142 

63.3 

0.445 

91.3 

0.121 

53.2 

0.669 

80.3 

0.246 

68.0 

0.001** 

29.9 

0.815 

71.7 
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TABLE  A-4-2  (Continued):  Level  12. 


Subtest 

Item 

Number 

Probability 

Percentage 

Correct 

V 

49 

0.009** 

65.4 

V 

52 

0.001** 

57.0 
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57 

0.048* 

63.3 
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0.209 

46 . 6 
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89.6 
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88 

0.009** 

73.9 
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0.001** 

63.5 
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53.8 
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W2 

44 

0.261 

85.2 

W2 

45 

0.001** 

72.2 

W2 

48 

0.005** 

60.5 

W2 

59 

0.004** 

55.9 

W3 
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Ml 
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Dr.  Joe  Hard.  Jr.  1  Mr  .  Thomas  A.  Warm 

AFHRL/MPMD  U.  5,  Coast  Guard  Institute 

Brooks  AfB,  TX  78836  p.  o.  Substation  18 

Oklahoma  City.  OK  73169 


